Applicant: EGNALetal. 
Appl. No. 10/705,896 

In the Claims: 

The following listing of claims will replace all prior versions and/or listings of claims in 
the application: 
Listing of Claims: 
What is claimed is: 
1.-17. (Canceled) 

18. (Currently Amended) A method of operating a video surveillance system, the video 
surveillance system including at least one sensing unit capable of being operated in first and 
second modes, the method comprising th e s t e ps of : 

operating a sensing unit in the first mode to scan for targets; 

processing image data from the sensing unit in arthe first mode to detect the presence of 
an interesting target; 

upon detecting an-the interesting target, operating the sensing unit in the second mode to 
track the interesting target and performing at least one of: to improve the quality of information 
about th e interesting target over the information that can be obtained in th e first mod e panning, 
tilting, and zooming, using super-resolution algorithms, digitally zooming into the image, 
employing different lenses or filters, or using a robotic actuator to obtain higher quality imagery 
of the interesting target than that obtained by operating the sensing unit in the first mode ; and 

processing image data from the sensing unit in a-the second mode to track the target by 
sending at least one of pan, tilt, and zoom commands to the sensing unit. 
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19.-20. (Canceled) 



2 1 . (Currently Amended) The method of Claim 1 8, wherein th e st e p of processing image 
data from the sensing unit in a first mode comprises- _th e s t e ps of : 

processing the image data with a vision module to detect and locate at least one targetby 
at least one of: 

1) geo-locating the at least one object in 3D space; or 

2) classifying pixels in the image information as background pixels or foreground 

pixels, using the foreground pixels to determine at least one blob, tracking at least one possible 
target based on at least one blob, determining when at least one blob merges or splits into one or 
more possible targets, and performing at least one of: 

a) filtering and predicting the image location of at least one of the possible 

targets, or 

b) calculating a 3D position of at least one of the possible targets if 

calibration information is available; 

determining if at least one predetermined condition has been violated by at least one 
target ; and 

classifying at least one possible target by target type into categories . 



-3- 



Applicant: EGNAL et al. 
Appl. No. 10/705,896 

22.-26. (Canceled) 

27. (Currently Amended) The method of Claim 24, further comprising tho step of providing 
summary statistics including at least one of the following: an estimate of the target's position in 
image space; the target's position in a world coordinate frame: the target's size in image space; 
the target's size in the world coordinate frame; the target's velocity in image space: the target's 
velocity in the world coordinate frame: the target's trajectory; the target's color; the color of at 
least one subcomponent of the target; a bounding box for the target in an image: or at least one 
estimate of location or size of at least one body part, in the case where the target is human to be 
usedaid in the stop of determining if at least one predetermined condition has been violated by at 
least one object. 

28. (Canceled). 

29. (Currently Amended) The method of Claim 18, wherein tho step of processing image 
data from the sensing unit in a-the second mode, if the second mode involves camera motion, 
compris e s tho steps of : 

segmenting the target from the background by detecting motion of the sensing unit and 
factoring it out; 

calculating color histogram information in a region of at least some pixels of image data 
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from the sensing unit; 

calculating an edge image; and 

using template matching, to integrate the results of the steps of motion segmentation, 
edge detection and color histogram formation, to obtain an estimate of the target's position. 

30. (Currently Amended) The method of Claim 29, wherein the step of processing image 
data from the sensing unit in a second mode further comprises the step of : 

generating summary statistics including at least one of an estimate of the target's position 
in image space; the target's position in a world coordinate frame: the target's size in image space: 
the target's size in the world coordinate frame; the target's velocity in image space: the target's 
velocity in the world coordinate frame; the target's trajectory; the target's color; the color of at 
least one subcomponent of the target; a bounding box for the target in an image; or at least one 
estimate of location or size of at least one body part, in the case where the target is human, based 
on various features of the target. 



31. (Canceled) 



32. (Currently Amended) The method of Claim 29, wherein the step of processing image 
data from the sensing unit in a second mode further comprises th e st e ps of : 

monitoring a length of time in which the system has been in the second mode; 
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monitoring the various summary statistics to detect if any target characteristic has met a 
predetermined rule for switching to the first mode; 

monitoring for any external stimuli indicating that the system should switch to the first 
mode; and 

if any predetermined conditions are met in any of the monitoring processes, switching 
back to the first mode. 

33. (Currently Amended) The method of Claim 18, further comprisin g the steps of : 

determining a best shot of the target while operating the visual module in either of the 
first and second modes; and 

delivering the best shot back to the user through the response module. 

34. (Currently Amended) The method of Claim 1 8, further comprisin g th e st e ps of : 
jf the target exits the field of view of the sensing unit, 

receiving, by the second sensing unit, information about a last known position of the 

target from the sensing unit; 

operating the second sensing unit in the first mode to scan for the target, focusing on a 

region near the last known position; 

processing image data from the second sensing unit in the first mode to detect the 

presence of a target; 
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upon detecting a target, operating the second sensing unit in the second mode to track the 

target; and 

processing image data from the second sensing unit in the second mode to track the target 

by sending at least one of pan, tilt, and zoom commands to the second sensing unit, 

wherein, if the target is not detected within a predetermined time period in the step of 

operating the second sensing unit in the first mode to scan for the target, focusing on a region 
near the last known position, the second sensing unit switches to scanning for targets in its entire 
field of coverage directing a second sensing unit to scan for and track the targ e t . 

35.-37. (Canceled) 

38. (Original) The method of Claim 18, wherein, if operating the sensing unit in the second 
mode involves camera motion, processing image data from the sensing unit in a second mode 
comprises: 

(a) performing corner detection to find interesting points; 

(b) searching for matches for interesting points in successive images; 

(c) assigning confidence values to the matches of the interesting points; 

(d) employing a robust averaging method to determine an estimate from a subset of the 
matches determined to have high confidence values; 

(e) warping a second successive image towards a first successive image; and 
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(f) subtracting the warped image from a current image to determine which pixels have 

moved. 

39. (Currently Amended) The method of Claim 38, wherein processing image data from the 
sensing unit in a second mode further comprises: 

performing steps (a)-(f) on reduced resolution images; and 

performing st e ps (a)-(f) on full resolution images, utilizing the results of performing steps 
(a)-(f) on reduced resolution images to r e duc e th e computational burden by limiting the s e arching 
to only a subset of the full resolution imag e s . 

40. (Currently Amended) The method of Claim 38, wherein processing image data from the 
sensing unit in a second mode further comprises: 

using three-frame differencin g to alleviate false motion . 

41 . (Original) A motion detection module to determine camera motion, comprising: 

a corner detection module to find interesting points; 

a search module to find matches for interesting points between successive images; 
a confidence value assignment module to assign confidence values to the matches of the 
interesting points; 

a robust averaging module to determine an estimate from a set of high confidence 
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matches; 

a warping module to warp one successive image to another successive image for direct 
comparison of the images; and 

a subtraction module, which subtracts the warped image from a current image to 
determine which pixels have moved. 

42.- 62. (Canceled) 

63. (New) The method of claim 18, further comprising at least one of the following: sending 
an e-mail alert; sounding an audio alarm; providing a visual alarm; logging an alert to a persistent 
medium; making a telephone call; transmitting a wireless message to a personal digital assistant; 
enabling a mechanical actuator; or providing position information to another sensing unit. 

64. (New) A video surveillance system, comprising: 

at least one sensing unit capable of being operated in first and second modes, the sensing 
unit operating in the first mode to scan for targets; 

a response module for, upon detecting the target, operating the sensing unit in the second 
mode to track the target and performing at least one of: panning, tilting, and zooming, using 
super-resolution algorithms, digitally zooming into the image, employing different lenses or 
filters, or using a robotic actuator to obtain higher quality imagery of the interesting target than 
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that obtained by operating the sensing unit in the first mode; and 

a computer system coupled to the sensing unit, the computer system receiving and 
processing image data from the sensing unit, detecting and tracking targets, and determining 
whether the sensing unit operates in the first mode or in the second mode based on the detection 
and tracking of targets and processing image data from the sensing unit in the second mode to 
track the target by sending at least one of pan, tilt, and zoom commands to the sensing unit. 

65. (New) The method of Claim 64, wherein the computer system further comprises: 

a vision module processing the image data to detect and locate at least one target by at 
least one of: 

1) geo-locating the at least one object in 3D space; or 

2) classifying pixels in the image information as background pixels or foreground 
pixels, using the foreground pixels to determine at least one blob, tracking at least one possible 
target based on at least one blob, determining when at least one blob merges or splits into one or 
more possible targets, and performing at least one of: 

a) filtering and predicting the image location of at least one of the possible 

targets, or 

b) calculating a 3D position of at least one of the possible targets if 
calibration information is available; 

determining if at least one predetermined condition has been violated by at least one 
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target; and 

classifying at least one possible target by target type into categories. 

66. (New) The method of Claim 65, wherein the vision module further provides summary 
statistics including at least one of the following: an estimate of the target's position in image 
space; the target's position in a world coordinate frame; the target's size in image space; the 
target's size in the world coordinate frame; the target's velocity in image space; the target's 
velocity in the world coordinate frame; the target's trajectory; the target's color; the color of at 
least one subcomponent of the target; a bounding box for the target in an image; or at least one 
estimate of location or size of at least one body part, in the case where the target is human to be 
used in determining if at least one predetermined condition has been violated by at least one 
object. 

67. (New) The method of Claim 64, wherein the vision module further comprises: 

a segmentation module to separate background pixels from foreground pixels; 

a blobizer to receive the foreground pixels from the segmentation module and to group 
connected foreground pixels into coherent blobs; 

a target tracker to process the coherent blobs, determine when they are targets, and to 
obtain position information for each target; 

a classifier to determine a target type for each target; and 
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a primitive generation module to generate summary statistics to be sent to the inference 
module. 

68. (New) The method of Claim 67, wherein the vision module generates summary statistics 
including at least one of an estimate of the target's position in image space; the target's position 
in a world coordinate frame; the target's size in image space; the target's size in the world 
coordinate frame; the target's velocity in image space; the target's velocity in the world 
coordinate frame; the target's trajectory; the target's color; the color of at least one 
subcomponent of the target; a bounding box for the target in an image; or at least one estimate of 
location or size of at least one body part, in the case where the target is human, based on various 
features of the target. 

69. (New) The method of Claim 67, further comprising an inference which, in the second 
mode, determines whether to switch back to the first mode by monitoring at least one of how 
long the system has been in the second mode, the position of the target in the image, the position 
of the target in a world coordinate frame, and one or more external stimuli. 

70. (New) The method of Claim 64, wherein the vision module determines a best shot of the 
target while operating the visual module in either of the first and second modes; and 

a response module coupled to the vision module, the vision module delivering the best 
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shot back to the user through the response module. 

71 . (New) The method of Claim 64, further comprising: 

a second sensing unit coupled to the sensing unit to receive information about a last 
known position of the target from the sensing unit; if the target exits the field of view of the 
sensing unit, the second sensing unit operating in the first mode to scan for the target, focusing 
on a region near the last known position and operating in the second mode to track the target 
upon detecting the target; and; 

the second sending unit coupled to the computer system, the computer system processing 
image data from the second sensing unit in the first mode to detect the presence of a target, 
processing image data from the second sensing unit in the second mode to track the target by 
sending at least one of pan, tilt, and zoom commands to the second sensing unit, 

wherein, if the target is not detected within a predetermined time period, the second 
sensing unit operates in the first mode to scan for the target, focusing on a region near the last 
known position, and switches to scanning for targets in its entire field of coverage. 

72. (New) A computer readable medium for use in a video surveillance system including at least 
one sensing unit capable of being operated in first and second modes, the computer-readable 
information storage medium storing computer-readable program code for causing the computer 
to perform the steps of: 
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processing image data from the sensing unit operating in the first mode to detect the 
presence of an interesting target; 

upon detecting the interesting target, operating the sensing unit in the second mode to 
track the interesting target and performing at least one of: panning, tilting, and zooming, using 
super-resolution algorithms, digitally zooming into the image, employing different lenses or 
filters, or using a robotic actuator to obtain higher quality imagery of the interesting target than 
that obtained by operating the sensing unit in the first mode; and 

processing image data from the sensing unit in the second mode to track the target by 
sending at least one of pan, tilt, and zoom commands to the sensing unit. 
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